Guskey Chapter 4- Evaluating Professional Development
Level One: Participants’ Reactions
Level One is primarily concerned with how participants regard their professional development experience including their reactions to formal presentations, workshops, courses, seminars, or institutes. This may also apply to their perceptions of curriculum development activities, study group experiences, or action research projects (p. 94).
Question 1- What questions are addressed?
Content Q’s- address the relevance, utility, timeliness of topics covered; they focus on the new knowledge, skills and understandings that are the basis for the program; can also relate to scope, magnitude, credibility, and practicality of the change required. *Participants generally express more positive perceptions toward the content if they have some say in determining what that content will be AND when content address specific problems and practical, relevant solutions that can be implemented immediately.
(Examples of content questions can be found on p. 95)
Process Q’s- related to the conduct and organization of the professional development experience- how were things done. In formal settings, process questions focus on program leaders or specific activities. In less formal settings, process questions generally related to the structure and format of the activities. Responses may vary wildly due to different learning styles. For this reason, it is wise to try to include a variety of activities and learning formats.
(Examples of process questions can be found on p. 96)
Context Q’s- generally related to the setting including facilities, lighting, technology, room size and temperature, etc. These factors can be a necessary prerequisite to a successful professional development session. It may also be beneficial to ask the participants to rate themselves as learners- this causes them to reflect more carefully on their conduct.
(Examples of context questions can be found on p. 98)
Question 2- How will the information be gathered? Usually information is gathered towards the end of the professional development session- the last 10-15 minutes. These forms are usually collected as people leave to ensure information is obtained from everyone who took part. The format of these evaluation forms can vary greatly. Most forms usually include both multiple choice items and open-ended questions. Things to consider when gathering information.
All participants vs. a sample of participants- how much data can you effectively analyze? Allowing the evaluator to choose who is sampled creates bias, so the best method is to randomize the participants, e.g. choose every 5th person.
Signing evaluation forms-requiring signatures can allow for follow-up or clarification but may increase reluctance to answer questions honestly. According to Guskey, the most important attributes of the evaluation form are honesty and candor.
Delayed evaluations- when participants evaluate a PD right after the session they tend to rush through the form providing inaccurate or missing data. Some evaluators allow participants to wait a few days to answer which can allow them to reflect more deeply on the experience. However, rarely do all participants take the time to complete the forms and return it. So, contingencies can be put in place to guarantee their submission, e.g. Credit is only given once the evaluation is submitted.
Alternative methods- focus groups and interviews can be alternative methods of evaluation, but they breech anonymity, so evaluators can present the answers of the participants in summary form. Reflective journals or professional learning logs, etc. can also be considered alternative forms and participants can be given guiding questions to direct their responses. These methods provide rich information from which to draw conclusions about the experience.
Ongoing evaluations- Gathering several evaluations over time in phases or in components provides more in depth information to be evaluated. It is important to be strategic about time and place of gathering this information, as both variables impact the type of information collected.
(Examples of many common evaluation forms can be found on pages 108-114)
Question 3- What is measured or assessed? Initially, the point is to measure participants’ initial satisfaction with the experience. There is no exemplar form and professional development administrators are encouraged to create one based on the specific professional development being offered. It is possible to evaluate professional development as “criterion referenced” or a “norm referenced”. Criterion-referenced refers to reactions to a professional development based on professional standards, specific criteria, or quality indicators. Norm-referenced refers to evaluation which occurs based on participants experience in relation to the other professional developments. Usually, it is better to create criterion-referenced evaluations as participants past professional development experiences can vary widely.
Question 4- How will the information be used? There are many uses for the information gathered including modifying content offered, choosing follow-up activities, guide further professional development activities, and evaluating the evaluation form itself. Each level of evaluation builds on itself. All evaluations can provide crucial data that inform all other levels of professional development.
“In addition to liking it, we would also hope that participants learned something from their professional development” (Guskey, 2000, p.83).
Focus
Measure the knowledge, skills, and perhaps attitudes that participants learned.
Purpose
To use information gathered to revise content, format, & organization of PD.
Assessments
· Customized to the specific learning at hand (not standardized) · Range from paper and pencil assessment to demonstration of skills. · Learning can be documented in many permanent products (reflections, portfolio, case study analysis).
Considerations
· Openness to “unintended learnings” · Use of a pre- and post-test to measure participant knowledge · Guskey recommendation to assess learner perception of knowledge acquired may seem counterintuitive, but he’s no dummy (p. 131).
Questions & Quick-n-Dirty Answers
Q: Why is it important to assess participants’ learning? A: 1)validates relationship between intended teaching and final learning, 2) indicates effectiveness, and 3) formative data to modify PD
Q: What questions are addressed at this level? A: Did they learn the planned objectives?
Q: What types of learning are assessed? A: 1) Cognitive, 2) Psychomotor, and 3) Affective
Q: How will information be gathered? A: Procedures should be based on intended learning goals
Q: What is measured or assessed? A: Knowledge, skills, attitudes, and/or beliefs
Q: How will the information be used? A: Used to guide improvements in content, format, and organization of PD.
Level Four: Participants’ Use of New Knowledge and Skills (Guskey, 2000) Level Four addresses whether professional development (PD) participants have reflected on what they learned through their experience and how the experience has resulted in changes in practice. Evaluation of Level Four can only be conducted after program participants have had a sufficient amount of time to reflect on ideas learned and fit newly acquired skills and knowledge to their own settings (see pg. 180 for discussion). Level Four, then, is addressing how PD affects practice.
Gathering and analyzing information about whether or not new practices are used, and how well they are used, are essential activities in evaluation professional development programs and activities.
Question 1- What questions are addressed?
1. Stages of Concern: The assessment of the affective domain involved when individual(s) change their practice. “Assessing participants’ concerns is crucial for both formative and summative evaluations of professional development” (p. 184). Generally, six stages of concern are assessed across four categories.
Category
Stage
Label
Description
Impact
6
Refocusing
Evidence of exploring benefits resulting from change indicated. Includes major alteration or adaptations of practice.
5
Collaboration
Evidence of coordinating with others regarding the change in practice indicated.
4
Consequence
Evidence of how the change is affecting student outcomes (how, which outcomes are affected), and necessary adaptations) indicated.
Task
3
Management
Evidence of a focus on processes and required to apply change in practice (best uses of information/resources) indicated.
Self
2
Personal
Evidence of a focus in on the demands of the change and the individual’s ability to meet those demands. Reward structures, decision making, and potential conflicts are assessed.
1
Informational
Evidence of thought regarding the details involved with implementing the change indicated. “need more information” stage.
Awareness
0
Awareness
Little concern about the change is indicated.
2. Levels of Use: Levels of use involve varying actions or non-actions related to a participant’s use of the new knowledge or skills gained through PD. Generally, there are three levels of non-use and five levels of use measured.
Category
Stage
Label
Description
Use
6
Renewal
Reevaluation of efforts indicated with needed alteration modification to practice.
5
Integration
Indicates an attempt made to collaborate with colleagues/disseminate practice.
4b
Refinement
Reflection indicated/Varies the use of practice within context to improve student outcomes.
4a
Routine
Pattern of use indicated but no thought/reflection on the practice.
3
Mechanical
Superficial day to day use of practice indicated. No reflection on practice.
Non-use
2
Preparation
Evidance of preparation for use.
1
Orientation
Information and resources for use investigated.
0
Non-use
No involvement and no interest in becoming involved in practice.
3. Differences in Practice: Concerns whether current practice is truly different from what was being done in the past. *PD must involve differences from what was being done in the past. Otherwise, teachers will not implement any changes because they cannot see what is different from what was already being done.
“Another, more complex issue is determining whether observed differences in practice are actually due to the professional development experience and not to other, extraneous factors.” (p. 187).
“Evidence gathered before serves to establish a baseline of part practices, whereas evidence gathered afterwards shows the extent of change or innovation” (p. 187).
Question 2- How will the information be gathered?
Instruments used to gather information in Stage Four seek to capture the presence or absence of critical indicators. Critical indicators answer the question, “What would we expect to see if effective implementation were taking place?” Identifying five or six critical indicators is recommended to effectively capture the nature of change occurring. Program evaluators identify critical indicators and then develop descriptive examples of what each critical indicator looks like when properly implemented. See page 190 of the text for an example. Once critical indicators are identified and defined, instruments can be found that assess the indicators. Examples of such instruments include but are not limited to:
· Direct observation · Interviews · Questionnaire · Focus Groups · Implementation logs · Participant Portfolios · Rubrics
It is often appropriate to use a comparison group to measure the validity of evaluation using these measures.
Question 3- What is measured or assessed?
The major issue being measured or assessed in Level Four is whether new knowledge and skills gained effect professional practice, and to what extent.
Key Question: What did participants learn through professional development that affected their professional practice?
Question 4- How will the information be used?
Information gathered in Level Four can be used both in a summative nature to evaluate overall impact or formatively to drive changes in the PD program.
Analysis can be used both formative and summative. Formatively, it can be used to facilitate better and higher quality implementation of new practices. Summatively, information on participants’ new knowledge and skills which help document implementation efforts.
Level 5
Level 5 is focused on wanting to know if the professional development program or activity was a benefit to students in any way. Did it result in improvements in students achievement, attitudes, perceptions, or behaviors?
Question 1- Why is it important to assess student learning outcomes?
"True purpose and ultimate goals was to provide educators with the professional knowledge and craft skills they need to help all students learn at high levels". (Guskey, 1999b)
-leads to changes in practice and improved results with students but must be accompanied with feedback, collaborative planning, and ongoing assistance.
-allowed professional development leaders to set expectations high and make their standards rigorous
-broadened perspectives on factors that influence professional development
-empowered professional developers to make what they do count
Did all students acquire the intended knowledge, skills, attitudes, beliefs, and behaviors? What were the unintended consequences and their effects?
Question 2-What types of Learning Are Assessed? Cognitive Outcomes-student academic achievements and accomplishments that students might be expected to acquire Affective Outcomes-student beliefs, feelings, dispositions, and attitudes that students might be expected to develop Psychomotor Outcomes- student behaviors, actions, or practices that we want students to acquire
Question 3- How will the information be gathered? Cognitive Outcome Information - standardized tests, teacher developed assessments, standardized performance assessments, grades, questionnaires, group tasks or activities, school records, and/or portfolios Affective Outcome Information - questionnaires and interviews Psychomotor Outcome Information - observations, questionnaires, interviews, and school records
Question 4-What other assessment issues should be considered? Validity -- especially construct validity Reliability Sampling --simple random and stratified random commonly used Importance of multiple measures -- assessments of change or improvement require multiple measure; formative evaluations require assessments of interim and final goals; evaluations must consider intended and unintended outcomes; and complex interactions among outcome measures necessitate the use of multiple measures. Disaggregation of data -- consider separately the evidence from different student subgroups Use of pre- and posttest to measure student learning outcomes Use of comparison groups to measure student learning outcomes Importance of timing -- when to collect information
Question 5 - How will this information be used?
Knowing who wants the information, what information they want, and how they plan to use it to help guide evaluation decisions.
Decisions based on accurate and meaningful information, appropriately analyzed and clearly presented, is the goal of every evaluation event.
Level One: Participants’ Reactions
Level One is primarily concerned with how participants regard their professional development experience including their reactions to formal presentations, workshops, courses, seminars, or institutes. This may also apply to their perceptions of curriculum development activities, study group experiences, or action research projects (p. 94).
Question 1- What questions are addressed?
Content Q’s- address the relevance, utility, timeliness of topics covered; they focus on the new knowledge, skills and understandings that are the basis for the program; can also relate to scope, magnitude, credibility, and practicality of the change required. *Participants generally express more positive perceptions toward the content if they have some say in determining what that content will be AND when content address specific problems and practical, relevant solutions that can be implemented immediately.
(Examples of content questions can be found on p. 95)
Process Q’s- related to the conduct and organization of the professional development experience- how were things done. In formal settings, process questions focus on program leaders or specific activities. In less formal settings, process questions generally related to the structure and format of the activities. Responses may vary wildly due to different learning styles. For this reason, it is wise to try to include a variety of activities and learning formats.
(Examples of process questions can be found on p. 96)
Context Q’s- generally related to the setting including facilities, lighting, technology, room size and temperature, etc. These factors can be a necessary prerequisite to a successful professional development session. It may also be beneficial to ask the participants to rate themselves as learners- this causes them to reflect more carefully on their conduct.
(Examples of context questions can be found on p. 98)
Question 2- How will the information be gathered?
Usually information is gathered towards the end of the professional development session- the last 10-15 minutes. These forms are usually collected as people leave to ensure information is obtained from everyone who took part. The format of these evaluation forms can vary greatly. Most forms usually include both multiple choice items and open-ended questions. Things to consider when gathering information.
All participants vs. a sample of participants- how much data can you effectively analyze? Allowing the evaluator to choose who is sampled creates bias, so the best method is to randomize the participants, e.g. choose every 5th person.
Signing evaluation forms-requiring signatures can allow for follow-up or clarification but may increase reluctance to answer questions honestly. According to Guskey, the most important attributes of the evaluation form are honesty and candor.
Delayed evaluations- when participants evaluate a PD right after the session they tend to rush through the form providing inaccurate or missing data. Some evaluators allow participants to wait a few days to answer which can allow them to reflect more deeply on the experience. However, rarely do all participants take the time to complete the forms and return it. So, contingencies can be put in place to guarantee their submission, e.g. Credit is only given once the evaluation is submitted.
Alternative methods- focus groups and interviews can be alternative methods of evaluation, but they breech anonymity, so evaluators can present the answers of the participants in summary form. Reflective journals or professional learning logs, etc. can also be considered alternative forms and participants can be given guiding questions to direct their responses. These methods provide rich information from which to draw conclusions about the experience.
Ongoing evaluations- Gathering several evaluations over time in phases or in components provides more in depth information to be evaluated. It is important to be strategic about time and place of gathering this information, as both variables impact the type of information collected.
(Examples of many common evaluation forms can be found on pages 108-114)
Question 3- What is measured or assessed?
Initially, the point is to measure participants’ initial satisfaction with the experience. There is no exemplar form and professional development administrators are encouraged to create one based on the specific professional development being offered. It is possible to evaluate professional development as “criterion referenced” or a “norm referenced”. Criterion-referenced refers to reactions to a professional development based on professional standards, specific criteria, or quality indicators. Norm-referenced refers to evaluation which occurs based on participants experience in relation to the other professional developments. Usually, it is better to create criterion-referenced evaluations as participants past professional development experiences can vary widely.
Question 4- How will the information be used?
There are many uses for the information gathered including modifying content offered, choosing follow-up activities, guide further professional development activities, and evaluating the evaluation form itself. Each level of evaluation builds on itself. All evaluations can provide crucial data that inform all other levels of professional development.
National Staff Development Council (Learning Forward): http://www.learningforward.org/standards/index.cfm
Level 2: Participant’s Learning
“In addition to liking it, we would also hope that participants learned something from their professional development” (Guskey, 2000, p.83).
· Range from paper and pencil assessment to demonstration of skills.
· Learning can be documented in many permanent products (reflections, portfolio, case study analysis).
· Use of a pre- and post-test to measure participant knowledge
· Guskey recommendation to assess learner perception of knowledge acquired may seem counterintuitive, but he’s no dummy (p. 131).
A: 1)validates relationship between intended teaching and final learning, 2) indicates effectiveness, and 3) formative data to modify PD
Q: What questions are addressed at this level?
A: Did they learn the planned objectives?
Q: What types of learning are assessed?
A: 1) Cognitive, 2) Psychomotor, and 3) Affective
Q: How will information be gathered?
A: Procedures should be based on intended learning goals
Q: What is measured or assessed?
A: Knowledge, skills, attitudes, and/or beliefs
Q: How will the information be used?
A: Used to guide improvements in content, format, and organization of PD.
Level Four: Participants’ Use of New Knowledge and Skills (Guskey, 2000)
Level Four addresses whether professional development (PD) participants have reflected on what they learned through their experience and how the experience has resulted in changes in practice. Evaluation of Level Four can only be conducted after program participants have had a sufficient amount of time to reflect on ideas learned and fit newly acquired skills and knowledge to their own settings (see pg. 180 for discussion). Level Four, then, is addressing how PD affects practice.
Gathering and analyzing information about whether or not new practices are used, and how well they are used, are essential activities in evaluation professional development programs and activities.
Question 1- What questions are addressed?
1. Stages of Concern: The assessment of the affective domain involved when individual(s) change their practice. “Assessing participants’ concerns is crucial for both formative and summative evaluations of professional development” (p. 184). Generally, six stages of concern are assessed across four categories.
2. Levels of Use: Levels of use involve varying actions or non-actions related to a participant’s use of the new knowledge or skills gained through PD. Generally, there are three levels of non-use and five levels of use measured.
3. Differences in Practice: Concerns whether current practice is truly different from what was being done in the past. *PD must involve differences from what was being done in the past. Otherwise, teachers will not implement any changes because they cannot see what is different from what was already being done.
“Another, more complex issue is determining whether observed differences in practice are actually due to the professional development experience and not to other, extraneous factors.” (p. 187).
“Evidence gathered before serves to establish a baseline of part practices, whereas evidence gathered afterwards shows the extent of change or innovation” (p. 187).
Question 2- How will the information be gathered?
Instruments used to gather information in Stage Four seek to capture the presence or absence of critical indicators. Critical indicators answer the question, “What would we expect to see if effective implementation were taking place?” Identifying five or six critical indicators is recommended to effectively capture the nature of change occurring.
Program evaluators identify critical indicators and then develop descriptive examples of what each critical indicator looks like when properly implemented. See page 190 of the text for an example. Once critical indicators are identified and defined, instruments can be found that assess the indicators. Examples of such instruments include but are not limited to:
· Direct observation
· Interviews
· Questionnaire
· Focus Groups
· Implementation logs
· Participant Portfolios
· Rubrics
It is often appropriate to use a comparison group to measure the validity of evaluation using these measures.
Question 3- What is measured or assessed?
The major issue being measured or assessed in Level Four is whether new knowledge and skills gained effect professional practice, and to what extent.
Key Question: What did participants learn through professional development that affected their professional practice?Question 4- How will the information be used?
Information gathered in Level Four can be used both in a summative nature to evaluate overall impact or formatively to drive changes in the PD program.
Analysis can be used both formative and summative. Formatively, it can be used to facilitate better and higher quality implementation of new practices. Summatively, information on participants’ new knowledge and skills which help document implementation efforts.Level 5
Level 5 is focused on wanting to know if the professional development program or activity was a benefit to students in any way. Did it result in improvements in students achievement, attitudes, perceptions, or behaviors?
Question 1- Why is it important to assess student learning outcomes?
"True purpose and ultimate goals was to provide educators with the professional knowledge and craft skills they need to help all students learn at high levels". (Guskey, 1999b)
-leads to changes in practice and improved results with students but must be accompanied with feedback, collaborative planning, and ongoing assistance.
-allowed professional development leaders to set expectations high and make their standards rigorous
-broadened perspectives on factors that influence professional development
-empowered professional developers to make what they do count
Did all students acquire the intended knowledge, skills, attitudes, beliefs, and behaviors? What were the unintended consequences and their effects?
Question 2-What types of Learning Are Assessed?
Cognitive Outcomes-student academic achievements and accomplishments that students might be expected to acquire
Affective Outcomes-student beliefs, feelings, dispositions, and attitudes that students might be expected to develop
Psychomotor Outcomes- student behaviors, actions, or practices that we want students to acquire
Question 3- How will the information be gathered?
Cognitive Outcome Information - standardized tests, teacher developed assessments, standardized performance assessments, grades, questionnaires, group tasks or activities, school records, and/or portfolios
Affective Outcome Information - questionnaires and interviews
Psychomotor Outcome Information - observations, questionnaires, interviews, and school records
Question 4-What other assessment issues should be considered?
Validity -- especially construct validity
Reliability
Sampling --simple random and stratified random commonly used
Importance of multiple measures -- assessments of change or improvement require multiple measure; formative evaluations require assessments of interim and final goals; evaluations must consider intended and unintended outcomes; and complex interactions among outcome measures necessitate the use of multiple measures.
Disaggregation of data -- consider separately the evidence from different student subgroups
Use of pre- and posttest to measure student learning outcomes
Use of comparison groups to measure student learning outcomes
Importance of timing -- when to collect information
Question 5 - How will this information be used?
Knowing who wants the information, what information they want, and how they plan to use it to help guide evaluation decisions.
Decisions based on accurate and meaningful information, appropriately analyzed and clearly presented, is the goal of every evaluation event.